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Final  Report 

The  Use  of  Tailored  Testing  with  Instructional  Programs 


The  primary  objective  of  the  project  was  to  investigate  issues  that  are 
related  to  the  implementation  of  tailored,  or  adaptive,  testing  in  the 
instructional  programs  environment.  These  issues  are  of  two  major  types: 

•  Those  related  to  the  design  of  a  computer  system  for  the 

administration,  scoring,  and  reporting  of  results  in  tailored 


tests . 

•  Those  related  to  the  psychometric  theory  that  is  the  foundation 
of  tailored  testing. 

To  address  these  issues  in  a  practical  and  realistic  way,  this  project 
involved  the  staff  at  the  Great  Lak.es  Naval  Training  Center  in  developing  and 
testing  a  computer  system  for  use  in  administering  tests  in  conjunction  with 
instructional  programs  at  the  base.  This  report  is  a  summary  of  the  knowledge 
gained  from  the  development  and  implementation  of  the  system. 

The  report  is  composed  of  three  sections.  The  first  describes  the 
instructional  environment  at  the  training  center  that  served  as  a  test  site 
for  the  tailored  testing  system.  The  second  section  describes  the  computer 
hardware  used  for  the  project  and  the  software  that  was  developed  for  the 
implementation  of  instructional  testing  at  Great  Lakes  Naval  Training 
Center.  The  third  section  describes  the  psychometric  research  that  was 
performed  as  part  of  the  project  and  summarises  the  results  of  the  research. 


Instruct :onai  environment 

Since  the  go  ’  of  this  pr  ject  was  to  evaluate  tailored  testing  for 


instructional  purposes  i  ~i  a  r.-„\  o  •,  -•  edwr  it  onal  environment,  arrangements 
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were  made  to  implement  tailored  testing  in  the  Radar  Technician  Training 
Course  in  the  Electronics  Technician  School  at  the  Naval  Training  Center  at 
Great  Lakes,  Illinois.  This  course  was  selected  for  use  because  it  was  taken 
by  a  relatively  large  number  of  trainees  (approximately  60  new  trainees 
started  the  course  each  week),  it  had  fairly  extensive  item  pools  available 
for  the  material  covered  by  the  course,  the  trainees  were  fairly  sophisticated 
about  computer  equipment,  and  systems  programming  support  was  available  from 
the  Naval  Education  and  Training  Program  Development  Center  Detachment  at 
Great  Lakes. 


Course  Description 

The  Radar  Technician  Training  Course  is  a  six  week  course  that  is  divided 
into  three  major  areas.  These  areas,  in  turn,  are  subdivided  into 
instructional  modules.  Table  1  presents  a  brief  outline  of  the  course, 
showing  the  major  areas  and  the  modules.  The  performance  of  trainees  in  the 
course  was  evaluated  using  examinations  covering  the  three  major  areas: 
power,  transmitter,  and  receiver.  The  trainees  were  required  to  receive  a 
score  equal  to  or  greater  than  64%  of  the  total  possible  on  each  exam  in  order 
to  proceed  through  the  course.  If  they  failed  a  test  they  were  given 
remediation  and  retested  over  the  same  material,  but  with  an  alternate  form  of 
the  test.  Trainees  who  failed  an  examination  three  times  were  dropped  from 
the  course. 

Ai'hough  the  testing  and  remediation  procedure  implied  a  self-paced 
instructional  strategy,  in  actuality  there  was  little  flexibility  in  the  rate 
i  -;hich  trainees  could  proceed  through  the  course.  At  most,  trainees  were 
i  !  .-wed  to  spend  three  extra  days  on  the  material  from  a  section  ot  the 


■  .'-i. 


.s  . 


Major 

Area 

Module 

Number 

Description 

4.1.1 

Introduction  to 

AN/SPS-10  Radar  Set 

Power 

4.1.2 

Primary  Power  Distribution 

4.1.3 

AC  Voltage  Regulator 

4.1.4 

DC  Power  Supply 

4.2.1 

Modulator 

Transmitter 

4.2.2 

Radar  Transmitter 

4.2.3 

RF  System 

4.3.1 

Radar  Receivers 

Receiver 

4.3.2 

Adapter,  Indicator 

4.3.3 

Video  Clutter  Suppressor 

4.3.4 

Antenna  System 

The  instructional  schedule  for  the  course  is  given  in  Table  2.  This 
table  gives  the  topic  number  for  the  class,  the  type  of  class,  the  class 
period,  the  topic  description,  and  the  ratio  of  students  to  instructor.  The 
first  three  digits  of  the  topic  number  refer  to  the  instructional  modules 
listed  in  Table  1.  The  fourth  digit  refers  to  specific  topics  within  the 
module.  The  class  period  indicates  the  number  of  hours  since  the  beginning  of 
training.  The  Radar  Technician  Training  Course  begins  at  hour  741  with  a 
three  hour  class  on  "Introduction  to  Radar  Systems"  and  ends  at  hour  980  after 
the  four  hour  performance  test  on  receivers.  Thus,  the  course  is  240  hours 
long,  arranged  into  30,  eight  hour  days. 


mm  m  i  u  p 
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Table  2 

Advanced  Electronics  Field  Master  Schedule 
Radar  Technicians  Training  Course 


TOPIC  NO. 

TYPE 

PERIOD 

TOPIC 

RATIO 

A. 1.1.1 

Class 

7A1 

Introduction  to  Radar  Systems 

20:1 

7A2 

20:1 

7A3 

20:1 

Class 

7AA 

GMT 

A. 1.1.1 

Class 

7A5 

Intro  to  Radar  Sys  (cont'd) 

20:1 

7A6 

20:1 

7A7 

20:1 

7A8 

20:1 

A. 1.1.2 

Class 

7A9 

Radar  Principles 

20:1 

750 

20:1 

751 

20:1 

Class 

752 

GMT 

A. 1.1. 2 

Class 

753 

Radar  Principles  (cont'd) 

20:1 

75A 

20:1 

755 

20:1 

A. 1.1.3 

Class 

756 

Introduction  to  AN/SPS-10 

20:1 

757 

20:1 

758 

20:1 

759 

20:1 

760 

20:1 

A. 1.1. 3 

Class 

761 

Intro  to  AN/SPS-10 

20:1 

762 

20:1 

763 

20:1 

76A 

20:1 

765 

20:1 

766 

20:1 

767 

20:1 

Class 

768 

GMT 

20:1 

A. 1.1. A 

Lab 

769 

AN/SPS-10  Lab 

6:1 

770 

A. 1.1. A  1J  Familiarization 

6:1 

771 

with  the  AN/SPS-10  Radar  Set 

6:1 

772 

A. 1.1. A  2 J  AN/SPS-10  Radar  Set 

6:1 

773 

and  AN/SPA-25  Indicator 

6:1 

77A 

operating  procedures 

6:1 

775 

776 


Class 


GMT 


TOPIC  NO. 


PERIOD 


TOPIC 


RATIO 


4. 1.2.1 


4. 1.2. 1 


4. 1.3.1 


4. 1.4.1 


4.1.2 


4. 1.2.2 


4. 1.3. 2 

4. 1.3.2 

4. 1.4.2 


4. 1.2. 2 


77 

Primary  Power  Distribution 

2 

78 

2 

79 

2 

80 

2 

81 

82 

2 

2 

83 

84 

GMT 

2 

85 

Pri  Power  Dist  (cont'd) 

2 

86 

87 

88 

2 

2 

2 

89 

2 

90 

91 

2 

2 

92 

GMT 

93 

94 

AC  Voltage  Regulator 

2 

2 

95 

96 

2 

2 

DC  Power  Supply 

2 

2 

2 

2 

01 

m 

AC  Voltage  Regulator  Lab 

wm 

DC  Power  Supply  Lab 

Primary  Power  Distribution  Lab 

4. 1.2.2  1J 

GMT 

09 

10 

1  1 

Written  Exam  (Power) 

2 

2 

1  1 

12 

13 

Primary  Power  Distribution  Lab 

z 

2 

14 

4. I. 2. 2  2 J 
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Table  2  (Continued) 


TOPIC  NO. 

TYPE 

PERIOD 

TOPIC 

RATIO 

Class 

816 

GMT 

4. 1.2. 2 

Lab 

817 

Primary  Power  Distribution  Lab 

6:1 

818 

4. 1.2. 2  2 J  (cont'd) 

6:1 

819 

6:1 

820 

6:1 

821 

6:1 

822 

6:1 

823 

6:1 

Class 

824 

GMT 

4.1.2  -  4.1.3 

Lab 

825 

Performance  Test  (PPD/ACVR) 

6:1 

826 

6:1 

827 

6:1 

828 

6:1 

4. 2. 1.1 

Class 

829 

Vacuum  Tube  TPG 

20:1 

830 

20:1 

831 

20:1 

Class 

832 

GMT 

20:1 

4. 2. 1.1 

Class 

833 

Vacuum  Tube  TPG  (cont'd) 

20:1 

834 

20:1 

835 

20:1 

836 

20:1 

837 

20:1 

338 

20:1 

839 

20:1 

340 

20:1 

4.2. 1.2 

Class 

841 

Modulator  (MPG) 

20:1 

842 

20:1 

843 

20:1 

844 

20:1 

345 

20:1 

346 

20:1 

4.2. 1.4 

Class 

847 

Solid  State  TPG 

20:1 

Class 

348 

GMT 

4. 2. 1.4 

Class 

349 

Solid  State  TPG  (cont'd) 

20:1 

350 

20:1 

351 

20:1 

852 

20:1 

4 . 2 . 2 . 1 

Cl  ass 

8  5  3 

Microwave  Devices 

20:1 

354 

20:1 

55  5 

20:1 

8 
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Receiver  Lab  A. 3. 1.6  1J 


GMT 

Receiver  Lab  (cont'd) 


Adapter  Indicator  Lab 
GMT 

Adapter  Indicator  Lab  (cont'd) 


Overall  Systems  Labs 


Overall  Systems  Lab  (cont'd) 


GMT 

Area  Review 


Written  Exam  (Rcvr) 
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Table 

2  (Continued) 

TOPIC  NO. 

TYPE 

PERIOD 

TOPIC 

RATIO 

975 

20:1 

Class 

976 

GMT 

4.3.1  -  4.3.2 

Lab 

977 

Performance  Test  (Rcvr) 

6:1 

978 

6:1 

979 

6:1 

980 

6:1 

Course  Exams 

The  three  tests  used  for  this  project  were  administered  at  hour  809 
(Power),  889  (Transmitter),  and  972  (Receiver).  Two  forms  of  each  of  the 
tests  were  available  for  use.  Table  3  summarizes  the  characteristics  of  each 
of  the  tests.  The  tests  are  quite  variable  in  quality  and  some  have  fairly 
low  reliabilities.  Each  of  the  tests  was  constructed  by  selecting  the  items 
from  an  item  pool  that  had  been  developed  for  each  content  area.  A 
description  of  the  item  pool  for  each  of  the  course  content  areas  is  given  in 
Table  4.  Since  the  items  were  sampled  from  the  item  pool  with  replacement, 
the  test  forms  frequently  had  items  in  common  and  not  all  items  in  the  pool 
were  used.  Therefore,  the  number  of  items  available  for  calibration  using 
item  response  theory  models  was  less  than  the  total  number  of  items  in  the 
pool.  The  fifth  column  of  the  table  indicates  the  number  of  items  available 
for  calibration  and  the  sixth  column  gives  the  sample  size  for  calibration. 
For  tests  A22  and  A23,  the  sample  sizes  varied  for  the  items  within  the  pool 
oecause  of  overlap  in  the  tests. 

I 


Module 


Number  of  Items 


Total 


Number  of 

Item  Calibrated  N 


1 

60 

2 

34 

3 

17 

4 

8 

1 

52 

2 

34 

3 

11 

1 

40 

2 

10 

3 

1 

4 

27 

System  Design 


Hardware 

The  system  design  for  the  computerized  adaptive  testing  project  at  Great  Lakes 
was  predicated  on  several  assumptions.  They  were: 

1.  Each  testing  station  should  be  capable  of  functioning  as  a 
stand-alone  test  administrator  for  reasons  of  system 

rel iabi 1 i ty . 

2.  The  results  of  a  test  would  have  to  be  accumulated  for  a 
class  so  that  class  reports  and  item  analyses  could  be 
generated . 

3.  The  system  should  be  able  to  administer  different  tests  to 


different  students. 


In  order  to  meet  the  assumptions  with  equipment  that  was  available  at  the 
start  of  the  project,  a  hierarchical  computer  structure  was  designed.  The 
system  had  three  levels:  (1)  the  testing  stations,  (2)  midlevel  computers  for 
collecting  results  from  eight  testing  stations  and  sending  them  to  a  central 
processor,  and  (3)  a  large,  top  level  computer  that  would  compile  data  and 
generate  the  reports.  This  configuration  is  shown  schematically  in  Figure  1. 

Each  testing  station  consisted  of  an  Ohio  Scientific  Challenger  computer 
with  two  floppy  disk  drives  and  the  OS  65U  VI. 3  operating  system  and  a 
Hazeltime  1420  terminal.  The  terminals  were  located  in  carrels  in  a  testing 
room.  The  computers  were  located  in  a  separate  room  that  was  under  the 
control  of  the  instructional  staff. 

Eight  of  the  testing  stations  were  connected  to  each  midlevel  computer. 
This  computer  was  also  an  Ohio  Scientific  Challenger  computer  with  two  floppy 
disk  drives  and  the  OS  65U  Vl.3  operating  system.  Although  four  midlevel 
computers  were  in  place  at  Great  Lakes,  only  22  testing  stations  were 
available.  Thus,  the  full  32  testing  station  capacity  of  the  system  was  not 
realized . 

The  four  midlevel  computers  transmitted  test  results  to  the  top  level 
computer.  This  computer  was  an  Ohio  Scientific  C-3B  microprocessor  with  48K 
random  disk  and  a  74  megabyte  (formated)  Winchester  hard  disk.  The  computer 
also  had  two  floppy  disk  drives  and  a  tape  backup.  Two  of  these  computers 
were  purchased  for  the  project.  One  served  as  a  backup  for  the  other  to 
insure  system  reliability.  The  two  top  level  computers  also  allowed  for 
potential  expansion  to  64  testing  stations. 


Software 


Two  test  administration  programs  were  developed  to  run  on  the  system. 

The  first  was  a  program  that  administered  a  fixed  set  of  test  items  to  each 
examinee  with  a  randomly  selected  order.  That  is,  each  examinee  received  the 
same  set  of  test  items,  but  the  order  of  the  items  was  randomly  determined  for 
each  individual.  This  program  was  to  be  used  at  the  initiation  of 
computerized  testing  to  collect  data  addressing  the  issue  of  whether  items 
function  the  same  when  administered  on  a  computer  screen  as  when  administered 
in  paper-and-pencil  form.  The  fixed  set  of  items  administered  by  the  computer 
was  the  same  item  set  used  on  the  regular  paper-and-pencil  tests. 

The  second  administration  program  developed  during  the  project  was  for 
the  adaptive  administration  of  the  course  tests.  This  program  was  based  on 
the  one-parameter  logistic  (Rasch)  item  characteristic  curve  model.  This 
model  was  selected  as  a  result  of  a  research  study  that  will  be  described 
later  in  this  report. 

Adaptive  testing  procedures  require  algorithms  for  selecting  items,  for 
estimating  ability,  and  for  terminating  the  testing  session.  The  program 
produced  for  this  project  used  maximum  information  item  selection,  maximum 
likelihood  ability  estimation,  and  a  sequential  probability  ratio  test 
(Reckase,  1983)  for  the  pass/fail  decision  to  terminate  testing.  Since  the 
maximum  likelihood  estimation  procedure  cannot  compute  ability  estimates  until 
both  correct  and  incorrect  responses  have  been  obtained,  a  fixed  stepsize,  up 
and-down  procedure  was  used  to  obtain  ability  estimates  before  both  types  of 
responses  were  available. 

After  the  final  ability  estimate  was  obtained  for  each  examinee,  the  item 
response  theory  theta  estimate  was  converted  to  an  estimated  true-score  based 
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on  Che  entire  item  pool  for  that  test.  The  estimated  true  score  was  the  score 
reported  to  both  the  examinee  and  the  instructor. 

Both  of  the  test  administration  programs  were  written  in  Ohio  Scientific 
BASIC  for  use  with  the  OS  65U  VI. 3  operating  system.  They  both  accessed  item 
pools  stored  on  floppy  disks  at  the  testing  station  computers. 

System  software  for  the  communications  among  computers  was  also  developed 
as  part  of  the  project.  Personnel  from  the  Naval  Education  and  Training 
Program  Development  Center  Detachment  at  Great  Lakes  assisted  with  the 
development  of  this  software.  This  set  of  programs  accumulated  test  results 
from  the  testing  stations  and  stored  them  on  the  hard  disk  at  the  top  level 
computers.  Other  software  developed  on  the  project  was  used  to  generate 
reports  for  the  instructor  using  this  information. 

The  computer  system  used  for  administering  the  test  at  Great  Lakes  was 
described  in  several  professional  papers  during  the  life  of  the  project.  The 
ret vences  for  these  papers  are  given  below. 

McKinley,  R.L.,  &  Reckase,  M.D.  (1983).  Adaptive  testing  in  a  military 
! rail,  ing  environment.  In  Proceedings  of  the  2  3th  Annual  Conference  of  the 
Military  Testing  Association  (pp.  118-123).  Pensacola,  Florida:  Naval 
Education  and  Training  Program  Development  Center,  Sanfley. 

•1  k'  ■  m  i  ey ,  R.L.,  &  Reckase,  M.D.  (1984,  April).  Implementing  an  adaptive 
t  c_s- ing  program  in  an  instructional  program  environment.  Paper  presented  at 
the  meeting  of  the  American  ’  --ational  Research  Association,  New  Orleans. 

Reck a  v ■  M.n.,  &  McKinley,  R.L.  v1984,  August).  The  use  of  adaptive  testing 
t  "im  rutt ional  programs.  Paper  presented  at  the  meeting  of  the  American 

.  k  )k"3l  Ass  sc  i at  ion,  Toronto. 

Research  Projects 

.e-e  performed  is  pan  of  this  research  project.  They 
■  •'»  ga  •  n  ini  or.. nt  i  >n  needed  seine  t  the  appropriate  item 
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response  theory  model  for  use  with  the  item  pools  available  for  the  project, 
(b)  a  study  to  determine  the  effect  of  mode  of  test  administration  (paper-and- 
pencil  or  computer)  on  the  operation  of  the  test  items,  and  (c)  a  survey  of 
the  attitudes  of  trainees  toward  taking  tests  on  a  computer. 

The  results  of  the  first  study,  on  the  appropriate  item  response  theory 
model  for  adaptive  testing  using  the  radar  technician  item  pool,  were  reported 
in  a  technical  report  and  a  convention  paper.  The  references  for  the  report 
and  paper  are  given  below. 

McKinley,  R.L.,  &  Reckase,  M.D.  (1983).  An  evaluation  of  one-and  three- 
parameter  logistic  tailored  testing  procedures  for  use  with  small  item 
pools .  (Research  Report  ONR  83-1).  Iowa  City,  Iowa:  The  American  College 
Testing  Program. 

McKinley,  R.L.,  &  Reckase,  M.D.  (1984,  April).  An  evaluation  of  one-and 
three-parameter  logistic  tailored  testing  procedures  for  use  with  small  item 
pools .  Paper  presented  at  the  meeting  of  the  National  Council  on  Measurement 
in  Education,  New  Orleans. 

This  study  used  a  two-stage  evaluation  plan  to  compare  the  ability 
estimates  yielded  by  computerized  adaptive  testing  procedures  based  on  the  one 
parameter  logistic  (1PL)  and  the  three  parameter  logistic  models  (3PL).  The 
first  stage  of  the  study  used  real  data,  while  the  second  stage  used  simulated 
data.  In  the  first  stage,  response  data  for  3000  examinees  were  obtained  for 
the  40  items  on  a  form  of  the  ACT  Assessment  Mathematics  Usage  Test.  The 
first  2000  cases  were  used  to  obtain  item  parameter  estimates  tor  both 
models.  Using  these  estimates,  1PL  and  3PL  tailored  tests  were  simulated 
using  the  response  data  for  the  remaining  1000  cases.  Roth  r  imp  .' .  :  .<■  ! 

adaptive  testing  procedures  employed  maximum  likelihood  ihili'v  •'s-it.;'i  :  and 
maximum  i  n  forma  t  ion  item  selection.  The  sets  of  ability  •  s'  ;  mu’  ; 


from  the  two  procedures  were  then  compared. 


In  the  second  stage  of  the  study,  response  data  for  3000  cases  were 
simulated  using  the  3PL  item  parameter  estimates  from  the  first  stage  as  true 
parameters.  True  abilities  were  selected  for  the  simulation  from  the  standard 
normal  distribution.  The  first  2000  cases  of  the  generated  data  were  used  for 
the  IPL  and  3PL  calibration  of  the  items,  and  the  remaining  1000  cases  were 
used  to  simulate  IPL  and  3PL  adaptive  tests.  The  ability  estimates  obtained 
from  the  two  procedures  were  compared  to  each  other  and  the  true  ability 
parameters . 

The  results  of  both  stages  of  the  study  indicated  that  the  IPL  and  3PL 
adaptive  tests  yielded  highly  correlated  ability  estimates,  and  there  was  not 
apparent  advantage  in  terms  of  ability  estimation  to  using  one  of  the  models 
over  the  other.  Because  the  IPL  procedure  was  less  expensive  to  use,  it  was 
the  recommended  model  for  this  application. 

The  result  of  the  second  study,  on  the  effect  of  mode  of  test 
administration,  was  reported  in  a  convention  paper.  The  reference  for  the 
paper  is  given  below. 

Atkivimn,  T.A.  (1935,  October).  An  investigation  of  the  effect  of 

udmi n i ster i ng  test  items  via  the  computer.  Paper  presented  at  the  meeting  of 

r he  Midwest  Educational  Research  Association,  Chicago. 

:  i  r.  uiy  compared  the  performance  of  86  items  from  the  item  pool  for 
i,-  A.-.  ? .  chni  '•  i  an  Training  Course  wh^n  administered  on  a  paper-and-penc  i  1 

i  :  o  i  •.  vh.-i,  ad.r.i  ni  stered  on  a  computer  screen.  The  responses  to  the 
r. ,  „  r  _  ;  i  r  s  •  anal  /.■•■d  us :  ng  the  uno-paramot  er  logistic  model  to  obtain 
i '  •  t  ;  /  n  tra-i>‘-  ••!  -s’  :ti  ■  ■  ;  |  r  m  each  of  the  administrations.  The  pairs  of 

■  :  .  r .  «.  •••  ■  n*  o  ana  :  v.-.o.f  ising  a  principal  components 
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technique  to  determine  which  items  had  difficulty  parameter  estimates  that  did 
not  fall  along  a  simple  linear  function. 

Of  the  86  items  analyzed,  26  were  found  to  exhibit  significant 
differences  in  performance  related  to  administrative  medium.  Eleven  items 
were  found  to  be  harder  when  administered  on  the  computer  screen,  and  15  were 
found  to  be  easier.  No  obvious  reasons  could  be  determined  for  the 
differences.  Several  hypotheses,  such  as  the  differences  being  related  to 
amount  of  verbal  material  in  an  item  or  item  format  were  eliminated.  Further 
work  needs  to  be  done  to  determine  the  cause  of  the  "mode  effect". 

The  third  research  study  carried  out  as  part  of  this  project  was  a  survey 
of  the  attitudes  of  trainees  at  Great  Lakes  toward  the  administration  of  tests 
by  computer.  A  sample  of  136  trainees  was  administered  an  attitude  survey 
immediately  following  the  administration  of  a  test  to  them  by  computer.  A 
copy  of  the  survey  is  included  in  an  appendix  to  this  report.  The 
distribution  of  responses  to  the  23  items  in  the  survey  is  presented  in 
Table  5. 


The  responses  generally  show  that  the  trainees  had  no  difficulty  taking 
the  test  on  the  computer  and,  in  fact,  62%  indicated  that  they  "enjoyed"  the 


computerized  test.  They  felt  that  the  instructions  to  the  test  were  clear  and 
that  the  pacing  of  the  test  was  not  too  fast  (the  items  were  untimed), 
although  they  indicated  that  taking  the  test  on  the  computer  was  faster  than 
for  a  paper-and-penc i 1  test.  There  was  little  indication  of  problems  with 
reading  the  terminal  screen  or  with  eye  fatigue.  The  trainees  had  no 
difficulty  finding  the  proper  keys  on  the  terminal  keyboard.  The  only 
negative  comment  concerned  examinees  being  unable  to  go  back  to  questions  once 
they  were  off  the  screen.  Sixty-eight  percent  indicated  that  they  were 
bothered  by  being  unable  to  go  back  to  previous  items.  Overall,  the  trainees 
had  no  difficulty  with  the  computerized  test  administrat ion. 


Summary  and  Conclusions 


This  report  describes  the  computerized  testing  system  that  was 
implemented  in  conjunction  with  the  Radar  Technician  Training  Course  at  the 
Naval  Training  Center  at  Great  hakes,  Illinois  and  the  research  that  was 
performed  using  the  system.  The  system  was  a  multilevel,  microprocessor-based 
computer  network  with  each  testing  station  capable  of  operating  as  a  stand¬ 
alone  test  administration  system.  The  system  was  used  to  administer  tests  in 
a  sequential,  fixed  length  forma'  for  the  purpose  of  gaining  informat’  ,  on 
the  effect  of  mode  of  administration  on  test  items. 

Although  software  was  developed  for  an  adaptive  administrat  i>n  ot 
'.tens,  the  system  w.i  n-'ver  used  i  >r  adaptive  '•■sting  because  el  re,  •  ■  : 

Hardware  failures.  These  failures  ,v.<  iia  r  1  l  >:  u  i  a  r  1  -  vex  '  r.g 
hardware  manufacturer.  Oh  ’  •'  So  ;  •  w  :  :  I  nc  .  .  wen  <<i'  ■  ;  has  ;  n.  i.  • 


project,  and  maintenance  was  difficult  to  obtain.  The  computer  system  used  is 
now  obsolete  and  cannot  be  obtained  through  any  source.  The  hardware  problems 
emphasize  the  need  for  developing  a  computerized  testing  system  using  hardware 
from  a  well  established  manufacturer. 

Despite  the  hardware  difficulties,  the  project  did  demonstrate  that  the 
system  design  was  viable  and  that  microcomputers  could  be  used  to  administer 
achievement  tests  in  an  instructional  program  environment.  The  survey 
research  showed  that  the  trainees  had  no  difficulty  taking  the  >est  on  the 
computer  terminals.  The  research  on  model  selection  showed  that,  for  the 
small  item  pools  available  for  this  course,  a  one-parameter  logistic  based 
testing  procedure  would  perform  adequately. 

One  research  finding  from  this  project  indicated  that  some  caution  should 
be  exercised  when  implementing  computerized  testing.  It  was  found  that  some 
test  items  performed  differently  when  administered  on  a  terminal  screen  than 
when  administered  on  a  paper-and-pencil  test.  Further  research  is  needed  to 
determine  the  kinds  of  items  that  are  affected  by  mode  of  administration  and 
whether  the  effect  will  make  the  item  easier  or  harder  than  the  paper-and- 
pencil  form. 

Overall,  this  project  has  shown  that  computerized  testing  can  be 
effectively  used,  but  that  system  reliability  should  be  a  major  factor  in  the 
development  of  such  a  system.  With  the  recent  advances  in  microcomputer 
hardware,  widespread  implementation  of  computerized  testing  in  support  of 
instruction  can  soon  be  expected. 
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PLEASE  COMPLETE  THE  FORM  BELOW  AND  RETURN  IT  TO  THE  TEST  ADMINISTRATOR  BEFORE 
LEAVING  THE  TESTING  ROOM. 

NAME _ 

LAST  FIRST  MIDDLE  INITIAL 

(Please  Print) 

SOCIAL  SECURITY  NUMBER  _  -  _  -  _ 

I 

DATE 


CLASS  NUMBER 


SHIFT 


QUESTIONNAIRE 


We  are  interested  in  your  reactions  to  this  new  form  of  testing.  Your  answers 
to  these  questions  will  provide  useful  information  about  this  testing 
process.  Please  read  each  question  carefully  before  responding. 

CIRCLE  THE  LETTER  OF  THE  APPROPRIATE  ANSWER  TO  EACH  ITEM. 

1.  I  have  used  a  computer.  .  . 

never  before. 

b.  once  or  twice. 

c.  occasionally. 

d.  frequently. 

2.  I  have  used  a  typewriter.  .  . 

a.  never  before. 

b.  once  or  twice. 

c.  occasionally. 

d.  frequently. 

3.  If  you  needed  assistance,  was  the  test  administrator  helpful? 

a.  yes 

b.  no 

c.  I  did  not  need  assistance 

4.  The  test  questions  appeared  on  the  screen.  .  . 

a.  too  quickly. 

b.  too  slowly. 

c.  at  about  the  right  speed. 


Overall,  the  computerized  test  was.  .  . 

a.  more  difficult  than  a  paper-and-pencil  test. 

b.  easier  than  a  paper-and-pencil  test. 

c.  about  as  difficult  as  a  paper-and-pencil  test. 

The  computerized  test  was.  .  . 

a.  faster  than  a  paper-and-pencil  test. 

b.  slower  than  a  paper-and-pencil  test. 

c.  about  the  same  as  a  paper-and-pencil  test. 

I  could  read  the  test  questions  on  the  screen.  .  . 

a.  with  great  difficulty. 

b.  with  some  difficulty. 

c.  easily. 

d.  very  easily. 

The  test  was  confusing.  .  . 

a.  only  during  the  instructions. 

b.  only  when  answering  questions. 

c.  during  both  instructions  and  answering  questions 

d.  not  at  all. 


My  eyes  felt  tired.  .  . 

a.  frequently. 

b.  occasionally. 

c.  once  or  twice. 

d.  not  at  all.  ‘.’’V/l 


How  clear  do  you  feel  the  computerized  instructions  were? 

a.  Very  clear  -  I  had  no  trouble  at  all  with  them 

b.  Clear  enough,  in  general  -  but  could  be  improved 

c.  Unclear  in  places  or  in  part 

d.  Very  unclear  and  confusing 

Did  you  have  enough  time  to  give  your  answers? 

a.  I  didn't  feel  rushed  or  pressured  at  all 

b.  I  felt  a  little  rushed  and  could  have  used  more  time 

c.  I  felt  rushed  and  pressured  for  time 

What  is  your  opinion  of  the  difficulty  level  of  the  questions? 

a.  They  were  too  difficult 

b.  They  were  about  right 

c.  They  were  too  easy 

I  understood  the  test  administrator's  instructions  and  introduction  to 
the  test. 

a.  Agree 

b.  Disagree 

c.  Undecided 

I  did  not  need  the  test  administrator's  instructions  in  order  to  take 
the  test. 

a.  Agree 

b.  Disagree 


c.  Undecided 


I  did  not  need  the  computerized  instructions  in  order  to  take  the  test 

a.  Agree 

b.  Disagree 

c.  Undecided 


I  had  difficulty  in  locating  the  proper  keys  on  the  keyboard. 

a.  Agree 

b.  Disagree 

c.  Undecided 


.  I  had  difficulty  in  pressing  in  right  keys. 

a.  Agree 

b.  Disagree 

c.  Undecided 


8.  I  felt  uneasy  about  taking  the  test  on  a  computer. 

a.  Agree 

b.  Disagree 

c.  Undecided 


9.  The  noise  from  the  computer  bothered  me  while  I  was  taking  the  test, 

a.  Agree 

b.  Disagree 

c.  Undecided 


Computerized  testing  is  more  impersonal  than  paper  and  pencil  testing, 

a.  Agree 

b.  Disagree 

c.  Undecided 


i 


I  enjoyed  taking  the  test  on  a  computer. 

a.  Agree 

b.  Disagree 

c.  Undecided 


I  was  bothered  by  not  being  able  to  change  my  answers  at  the  end  of  the 
test . 

a.  Agree 

b.  Disagree 

c.  Undecided 


23.  My  eyes  were  tired  by  the  end  of  the  test. 

a.  Agree 

b.  Disagree 

c.  Undecided 
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Code  22? 

POP  N .  Ouincy  Street 
Arlington,  VA  22217--5000 

Dr.  Mans  Pronbag 
University  of  Leyden 
Education  Research  Center 
Bo  e  r  h  a  a  v  e  1  a  a  n  2 
a?-.U  fm  Leyden 
The  NFTMFPLA *'DS 

CTP/MeGraw-Hi 1 1  Library 
25P0  Garden  Road 
Monterey,  CA  9?91j0 

Dr.  Dattprasad  Divgi 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.n.  Box  1626P 
Alexandria,  VA  22302- 026P 

Dr.  Hei-Ki  Dong 
Ball  Foundation 
a00  Roosevelt  Road 
Building  C,  Suite  206 
Cl  on  Ell yn ,  IL  601  ” 

Defense  Technical 

Information  Centen 
Cameron  Station,  Bldg  5 
Alexandria,  VA  22^14 
Attn:  TC 
f 1 2  Co  pi es) 

Dr.  Stephen  Dunbar 
Lindquist  Center 
for  Measurement 
Ur  i  ver  s i  ty  of  Iowa 
Iowa  City,  IA  5??U? 

Dr.  James  A.  Far 1 es 

Air  Force  Human  Resources  Lab 

Brooks  AFP,  T>" 


Dr.  Kent  Eaton 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  227N? 

Dr.  John  M .  Eddins 
University  of  Illinois 
252  Engineering  Research 
Laboratory 

10?  South  Mathews  Street 
Urbana,  TL  61801 

Dr.  Susan  Embretsor 
University  of  Kansas 
Psychology  Department 
Lawrence,  KB  660UP 

ERTC  Facility-Acquisitions 
Rugby  Avenue 
Bethesda,  MD  200 1 M 

Dr.  Benjamin  A.  Fairbank 
Performance  Metrics,  Tnc. 

5E2B  Callaghan 
Suite  225 

San  Antonio,  TX  7B22P 

Dr.  Leonard  Feldt 
Lindquist  Center 
for  Measurement 
University  of  Towa 
Iowa  City,  IA  52242 

Dr.  Richard  L.  Ferguson 
American  College  Testing 
Program 
P.0.  Box  1 6P 
Iowa  City,  I A  F224P 

Dr.  Gerhard  Fischer 
Li ebiggess"  5/7 
A  10 IP  Vienna 
AUSTRIA 

Prof.  Donald  Fitzgerald 
University  of  New  England 
Department,  of  Psychology 
Armidale,  New  South  Wales  ?~r>l 

AUSTRALIA 

Mr  .  Paul  Foley 

Navy  Personnel  RM1  rent.er 

San  Diego,  CA  D2152 
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Dr.  Leonard  Kroeker 
Navy  Personnel  RStD  Center 
San  Diego,  CA  92152 

Dr.  Michael  Levine 
Educat icnal  Psychology 
210  Fducation  Bldg. 

University  of  Illinois 
Champaign,  TL  61801 

Dr.  Charles  Lewis 
Faculteit  Sociale  Wetenschappen 
Ri jksuniversiteit  Groningen 
Oude  Boter ingestraat  2 7 
9712GC  Groningen 
The  NETHERLANDS 

Dr.  Robert  Linn 
College  of  Fducation 
University  of  Illinois 
Urbana,  IL  M801 

Dr.  Robert  Lockman 
Center  for  Naval  Analysis 
4U01  Ford  Avenue 
P.o.  Pox  1 6?6P 
Alexandria,  V A  22202-0268 

Dr.  Frederic  M.  Lord 
Educational  Testing  Service 
Princeton,  NJ  085^1 

Dr.  James  Lumsden 
Department  of  Psychology 
University  of  Western  Australia 
Ned  lands  W.A.  fP09 
AUSTRALIA 

Dr.  Will1 am  L.  Maloy 
Chief  of  Naval  Fducation 
and  Training 
Naval  A i r  Star  ion 
Pensacola,  FL  '2^08 

Dr  .  lory  V.irec 
Stop  •' '  -F 

F.  due  at  '  --r  ?i  Testing  Service 
pr-T'  >*-on  .  NJ  r one  ! 

Dr  .  '  •  -  n  "art  i  n 

A*"ny  ■  >-^h  Tnst.  itute 


Dr.  James  McBride 
Psychological  Corporation 
c/o  Harcourt,  Brsc°, 

Javanovich  Inc. 

1250  West  6th  Street. 

San  Diego,  CA  02101 

Dr.  Clarence  McCormick 

HQ,  MEPC0M 

MEPCT-P 

2500  Green  Pay  Road 
North  Chicago,  IL  rno6iJ 

Mr.  Robert  McKinley 

University  of  Toledo 

Department  of  Educational  Psychology 

Toledo.  OH  U2606 

Dr.  Barbara  Means 
Human  Resources 

Research  Organization 
1100  South  Washington 
Alexandria,  VA  ???1H 

Dr.  Robert  Mislevy 
Educational  Testing  Service 
Princeton,  NJ  088U1 

Headquarters,  Marin0  Corps 
Code  MPT -20 
Washington,  DC  20  2P0 

Dr.  W.  Alan  Nirewander 
University  of  Oklahoma 
Department  of  Psychology 
Oklahoma  City,  OK  "pObO 

Dr.  William  F.  Nordbrock 
FMC-ADCO  Box  25 
AP0,  NY  09710 

Dr.  Melvin  R.  Novick 
206  Lindquist  Center 
for  Measurement 
University  of  Towe 
Iowa  City,  ta  c22 a? 

Director,  Mnnpower  and  Personnel 
Laboratory , 

NPRDC  (Code  06) 

San  Diego,  CA  0?15? 
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Library.  NPRDC 

Code  PP01L 

San  Diego,  CA  92 1  5? 

Comn-’nding  Officer. 

Naval  Research  Laboratory 
Code  P6PY 

Wasting  ton,  DC  ,?0Son 

Dr.  James  Olson 
WIC  AT,  Inc. 

1 R  7  5  South  State  Street 
Orem,  IJT  RACE? 

Office  of  Naval  Research, 

Code  m?PT 
poo  N.  Ouincy  Street 
Arlington,  VA  ???17-5090 
(6  Copies) 

Special  Assistant  for  Marine 
Corps  Matters, 

ONR  Code  OOMC 
poo  N.  Ouincy  St. 

Arlington,  VA  22217-50D0 

Dr.  Judith  Orasanu 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  ??•” 

Wayne  m.  Patience 
American  Council  on  Education 
CEP  "esting  Service,  Suite  ?0 
0  n  f  Pu  po  r:  t  ('ire  1  e  ,  MW 
Washington,  DC  POO^h 

Dr.  Janes  Paulson 
Department  of  Psychology 
Portland  State  University 
P.0.  Pox  ooi 
Porv  1  and  ,  OR  97VT 

P-  .  Roger  Pennell 
Air  Force  Human  Resources 
Laboratory 
Lowry  AFP.  CO  P.nptip 

Dr.  t'ark  D.  Reckas« 

ACT 

P.  0.  Pox  1 6e 
Town  City,  TA  c??>n 


Dr .  Ma 1 colm  Ree 
AFHRL/MP 

P-ooks  AFB ,  TX  780 RS 

Dr.  Carl  Ross 
CMFT-PDCD 
Building  00 

Oeet  Lakes  NTC ,  IL  tnOW 
Dr.  .1.  Ry.an 

Department  of  Education 
U  iversity  of  South  Carolina 
Columbia,  SC  29 2.0° 

Dr.  Fumikc  Samejima 
Department  of  Psychology 
University  of  Tennessee 
Knoxville,  Tf!  V91* 

Mr.  Dr»w  Sands 
NPRDC  Code  6? 

San  fiego,  OA  9? 15? 

Dr.  Robert  Sasmor 
Army  Research  Tnstitute 
5001  Eisenhower  Avenue 
Alexandria,  VA 

Dr.  Mary  Schrat.z 

Navy  Personnel  RAD  Center 

San  Diego,  CA  9? IP? 

Dr.  W.  S*\eve  Sellman 
OASPCMRAAL 1 
?B26°  The  Pentagon 
Washington,  DC  PO’OI 

Dr.  Kazuo  Shigemasu 
v_p_pii  Kugenuma-Kaigan 
Fujusawa  2r l 
JAPAN 

Dr  .  Wi 1  1 i am  Sims 
Center  for  Naval  Analysis 
mini  Ford  Avenue 
P.O.  Pox  1  A  ?(;. p 
Alexandria.  VA  ??Dp— opf  p 
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Dr  .  fi .  'wul  1  hcp  D-i  na  i  ko 
Manpower  Rest  arch 

and  ;  tv  is or y  Services 
Smithson: an  ;nst itut ion 
yo  i  Ncrt  y_  pjt  t.  Sr  r  <°t, 

»;•  v,  •  :  .  VA  •’  '  •'  1  A 

Dr.  Richard  .cor>'iiS‘'n 
Navy  Personnel  RAD  C-nttT 
S-n  Dingo,  ri 

Dr  .  p;  ;  f packman 
■’ni  vers:  t  y  of  Missouri 
Department  of  Statistics 
Co  I  imb  i  a  .  tffi  f~ C>?C  1 

Dr  .  Mar  tho  St  ook)  ns 

!•' :i u'- ion, a]  Testing  Service 

tv  iric«  ton  .  U.I 

Dr.  peter  Ctoloff 
f>  r;- ,,r  for  Naval  Analysis 
PC1'  Nor*’  h  Pe;>u- f'gard  Street 
A'  ex  andri  a .  VA  :'?•>  1 1 

Dr  .  W:  1  1  '  cm  ft.  out 
Drives:  v  of  Illinois 
•  cf  Mathematics 

L’rbana  1,  <"•  1  « 0  1 

M  i ; .  ’ •  :  1  :"t  i  ckl  and 
AP  'vp.vtA 

h  •  1  A  2  P « -  r )  \  ,jg  pr 

-rV  1  f)i(*  <-.r  •  r 

’>•  .  "f  r'w -»n  •  n;it  hnn 

;  .  r,  ■  f  t *" i r  ^n-i 

:  •  •  .=•.<•  r,r,  ur,<.u 

P  nf  r'lu  •  »*  ’  in 

1  I  f!  i  /  *”  r'  i  V  %’  f  O  ••  r'  |  J  *•*  P  t  ^ 


tiller 


Dr.  Maurice  Tatsuoka 
2PC  Fducation  Pldg 
Pin  n.  Sixth  ft. 

Champaign,  IL  c  i upc 

Dr.  David  Thissen 
Department  of  Psychology 
University  of  Kansas 
Lawrence,  KS  660UH 

Me.  Gary  Thnm3sson 
University  of  Tllinols 
Educational  Psychology 
Champa ign  .  TL  61PP0 

Dr.  Pobert.  Tsutakawa 
The  Fred  Hutchinson 

Caned-  Research  Center 
Division  of  Public  Health  Pci. 

1 1  PFi  Columbia  Street 
Seattle,  W A  ORIOn 

Dr.  Ledyard  Tucker 
University  of  Illinois 
Department,  of  Psychology 
60'  E.  Daniel  Street 
Champaign,  IT.  C1°?0 

Dr.  Vern  VI.  Ilrry 
Personnel  RAD  Center 
Office  of  Personnel  Management 
1  on  p  t'  ^  rf.pt"y  ^  fjty 

Washington,  DC  PC  HIS 

Dr.  David  Vale 
A s sc s sn  ‘-r:’  c'v  s.  t  <--r,  s  Co  r  p  . 

~>P  ’7  "ni  versify  Avenue 

Suite  f1n 

St  pau  1  U?!  Do  "  1 11 

Dr  .  Fr :  r V i  ci  no 

Navy  p.  rsonni'!  RAD  Center 

fan  r«  v.viD? 

Dr.  Howard  V.'ainer 

Division  of  Psychological  Studies 

Flir  a*  ion  a  I  Tt  .ding  Service 
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Dr.  Ming-Mei  Wang 
Lindquist  Center 
for  Measurement 
University  of  Iowa 
Iowa  City,  TA  52242 

Mr.  Thomas  A.  Warm 
Coast  Guard  Institute 
P.  0.  Substation  IP 
Oklahoma  City,  OK  73169 

Dr.  Brian  Waters 
Program  Manager 
Manpower  Analysis  Program 
HumRRO 

1100  s.  Washington  St. 
Alexandria,  V A  ?2j14 

Dr.  David  J.  Weiss 
Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  Ml!  55455 

Dr.  Ronald  A.  Weitzman 
NPS,  Code  54Wz 
Monterey,  CA  92152 

Major  John  Welsh 
AFURL/MOAN 

Brooks  AFP,  TX  7 P22? 

Dr.  Rand  P.  Wilcox 
University  of  Southern 
Cal i forn i a 

Depart-ment  of  Psychology 
Los  Angeles,  CA  90007 

German  Military  Representative 
ATTN:  Wolfgang  Wild^grube 
St  re i tkr  aef team t 
P-  c'! 00  Bonn  ? 

4000  Brandywine  Street.,  NW 
Washington,  DC  20016 

Dr.  Bruce  Williams 
Department  of  Educational 
Psychology 

University  of  Illinois 
IJrbana.  TL  M»P1 
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Dr.  Hilda  Wing 
Army  Research  Institute 
5001  Eisenhower  Ave. 
Alexandria,  VA  2??,?? 

Dr.  Martin  F.  Wiskoff 
Navy  Personnel  RAD  Center 
San  Diego,  CA  9215? 

Mr.  John  H.  Wolfe 

Navy  Personnel  RAD  Center 

San  Diego,  CA  9215? 

Dr .  George  Wong 
Biostatistics  Laboratory 
Memorial  Sloan-Ketteri ng 
Cancer  Center 
1275  York  Avenue 
New  York.  NY  10021 

Dr.  Wendy  Yen 
CTB/McGraw  Hill 
Del  Monte  Research  Park 
Monterey,  CA  93940 
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